Evaluation of a Temporal-Abstraction Knowledge-Acquisition Tool

We describe the design and evaluation of a graphical knowledge-acquisition (KA) tool for entering the knowledge required by the RÉSUMÉ system. RÉSUMÉ is an implementation of the knowledge-based temporal-abstraction problem-solving method, which forms high-level concepts and patterns from raw time-oriented data. We designed the KA tool using the Protégé framework. The study evaluated the usability of the KA tool for entry, by domain experts and knowledge engineers, of previously elicited knowledge in three medical domains. We found that understanding the RÉSUMÉ system required 6 to 20 hours (median: 15 to 20 hours); learning to use the KA tool required 2 to 6 hours (median: 3 to 4 hours). Entry times for physicians varied by domain: 2 to 20 hours for the growth monitoring domain (median: 3 hours), 6 and 12 hours for the diabetes-care domain, and 5 to 60 hours for a protocol-based care domain (median: 10 hours). A speedup of up to 25 times (median: 3 times) was demonstrated for all participants when the KA process was repeated. On their first attempt at using the tool to enter the knowledge, the knowledge engineers recorded entry times similar to those of the expert physicians’ second attempt at entering the same knowledge. In all cases, RÉSUMÉ, using the knowledge entered via the KA tool and a benchmark set of data, generated abstractions that were almost identical to those generated using the same knowledge when entered manually (as text files) by knowledge engineers experienced in working with the RÉSUMÉ system.